                          Installation Notes
					QLogic's ESX FCoE Offload Driver
   
	                        QLogic Corporation

              Copyright (c) 2015 QLogic Corporation
                           All rights reserved


Table of Contents
=================

  Introduction
  OS'es Supported
  Load and Run Necessary iSCSI Software Components
  BNX2FC Driver Dependencies
  BNX2FC Module Parameters
  Unloading and Disabling Driver
  Listing FCoE Adapters and Creating Interfaces:
  Useful Supplementary Commands
  Maximize FCoE Offload Connections
  Caveats
  

Introduction
============

This file describes the bnx2fc driver for the QLogic 10 Gigabit
Ethernet FCoE Adapter 1000/10G/20G Mbps PCIe CNIC Network Controllers.
The bnx2fc driver enables FCoE offload on QLogic 10 Gigabit Ethernet
FCoE Adapter family of devices in ESX5.0 and later releases.


OS'es Supported
================
	ESX 5.0, ESX5.1, ESX5.5


Load and Run Necessary FCoE Software Components
================================================
QLogic's FCoE Offload software suite consists of 3 kernel modules, bnx2fc,
cnic and bnx2x.

Required software components can be loaded either manually or through system
services (JumpStart Plugin).

Manual loading :
----------------
   # vmkload_mod bnx2fc

NOTES: unloading bnx2fc driver is not officially support in ESX environment

Disabling bnx2fc driver:
------------------------
   # esxcfg-module -d bnx2fc


BNX2FC Driver Dependencies
==========================

The driver uses library functions in libfcoe, libfc and cnic. It is required
to load these library modules before attempting to load the driver or operation
will fail with unresolved symbolsi error. Also caution should be taken to not
load mis-matched drivers, this could lead to disabling FCoE functionality as
a result of HSI (Host Software Interface) differences.


BNX2FC Module Parameters
========================

Following optional parameters can be supplied as a command line arguments
to 'vmkload_mod' command or set as persistent configuration for auto loading.
QLogic does not expect or recommend the end user to experiment unless they
are clear about the impact of these parameters values.

All module parameter set through 'esxcfg-module -s' will only be effective
after reboot.

--------------------
bnx2fc_debug_level :
--------------------
Description: "Bit mask to enable/disable debug logs", debugging parameter

Defaults: 0
Valid Values: bit mask, currently lower 8-bits are supported

#define LOG_IOERR	0x00000001	/* scsi cmd error, cleanup */
#define LOG_SESS	0x00000002	/* Session setup, cleanup, etc' */
#define LOG_DEV_EVT	0x00000004	/* Device events, link, mtu, etc' */
#define LOG_SCSI_TM	0x00000008	/* SCSI Task Mgmt */
#define LOG_LPORT	0x00000010	/* lport related */
#define LOG_RPORT	0x00000020	/* rport related */
#define LOG_ELS		0x00000040	/* ELS logs */
#define LOG_FRAME	0x00000080	/* fcoe L2 frame related logs*/
#define LOG_INIT	0x00000100	/* Init logs */
#define LOG_DISCO	0x00000200	/* Link discovery events */
#define LOG_TIMER	0x00000400	/* Timer events */
#define LOG_INFO	0x00000800	/* Informational logs, e.g. device MFS,
					 * MAC address, WWPN, WWNN */
#define LOG_MP_REQ	0x00001000	/* Middle Path (MP) related */
#define LOG_ERROR	0x00002000	/* log non-fatal errors */
#define LOG_UNSOL	0x00004000	/* unsolicited event */
#define LOG_NPIV	0x00008000	/* unsolicited event */
#define LOG_FCP_ERR	0x00010000	/* log fcp errors */
#define LOG_VLAN	0x00020000	/* log vlan info */
#define LOG_INITV	0x40000000	/* Init logs */
#define LOG_SESSV	0x80000000	/* Session setup, cleanup, etc' */
#define LOG_ALL		0xffffffff	/* LOG all messages */

	0x00000001 - scsi cmd error, cleanup
	0x00000002 - Session setup, cleanup, etc'
	0x00000004 - Device events, link, mtu
	0x00000008 - SCSI Task Mgmt
	0x00000010 - lport related
	0x00000020 - rport related
	0x00000040 - ELS logs
	0x00000080 - fcoe L2 frame related logs
	0x00000100 - Init logs
	0x00000200 - Link discovery events
	0x00000400 - Timer events
	0x00000800 - Informational logs, e.g. device MFS, MAC address, WWPN, WWNN 
	0x00001000 - Middle Path (MP) related
	0x00002000 - log non-fatal errors
	0x00004000 - unsolicited event
	0x00008000 - NPIV event
	0x00010000 - log fcp errors
	0x00020000 - log vlan info
	0x40000000 - Init logs
	0x80000000 - Session setup, cleanup, etc'
	0xFFFFFFFF - LOG all messages

--------------------
bnx2fc_enable_r_a_tov :
--------------------
Description: "To enable/disable user defined R_A_TOV" 

Defaults: 0
Valid Values: 0 - Use System Default.
	      1 - Use the value of bnx2fc_r_a_tov module parameter as the R_A_TOV value.

--------------------
bnx2fc_r_a_tov :
--------------------
Description: "The user defined value of R_A_TOV" 

Defaults: 10 (in seconds)
Valid Values: 10-120 (in seconds)

--------------------
bnx2fc_vmk_pool :
--------------------
Description: "To enable/disable vmk_memPool"

Defaults: 1
Valid Values: 1 - vmk_MemPool mode.
			0 - non vmk_MemPool mode.

--------------------
bnx2fc_enable_cvl_fix :
--------------------
Description: "To enable/disable insertion of VxPort ID CVL"

Defaults: 1
Valid Values: 1 - Validate FIP cvl.
			0 - No need to Validate FIP cvl.

Unloading and Disabling Driver
==============================

Unloading the driver is not supported in ESX, however bnx2fc can be disabled and
inhibited from loading during boot

	#esxcfg-module -d bnx2fc


Listing FCoE Adapters and Creating Interfaces:
==============================================
1. List open-fcoe based FCoE adapters
	# esxcli fcoe nic list

2. Creating FCoE Interface:
	# esxcli fcoe nic discover -n|--nic-name=<vmnic#>

3. Destroy FCoE interface
	# esxcli fcoe nic disable -n|--nic-name=<vmnic#>

	Note: This requires server reboot to take affect

4. List FCoE interfaces /adapters
	# esxcli fcoe adapter list

	Displays Source / FCF / VNPort MAC Addresses, VLAN ID and vmnic#

5. List Storage Adapters and FCoE port state
	# esxcfg-scsidevs -a

	Sample output -
		vmhba34 bnx2fc            link-up   fcoe.100000101896d161:200000101896d161  () Software FCoE
		vmhba35 bnx2fc            link-up   fcoe.100000101896d163:200000101896d163  () Software FCoE



Useful Supplementary Commands
=============================

Rescan Adapter:
---------------
	# esxcfg-rescan <vmhba#>


List all Discovered FC LUNS:
----------------------------
	# esxcfg-mpath -b



Maximum FCoE Offload Connections
================================
With default driver parameter set which includes 256 outsanding commands, bnx2fc
can offload 128 connections.


Caveats:
========

1. esxcfg-fcoe is not supported
-------------------------------
	This CLI is not supported in ESX5.0 per Vmware FCoE Dev team.
	Please use 'esxcli fcoe' instead.

2. FCF re-discovery not happening after 2.5 * FKA
-------------------------------------------------
	Current libfcoe implementation does kick start re-discovery after
	missing 3 Discovery Advertisements.

3. iSCSI and FCoE offload coexistence
-------------------------------------------------
        Currently, there are no driver/firmware/hardware restrictions with
        iSCSI and FCoE connections being offloaded at the same time.  But
        please, note that having both offloads running at the same time
        is not officially supported by VMware.  Currently, VMware only
        officially supports either an offloaded iSCSI or an offloaded 
        FCoE connection on the same device because of restrictions in
        the driver certification test suite.  Please contact VMware for
	more information on this limitation.

3.  Default to VLAN 1002 if FIP VLAN discovery fails
-------------------------------------------------
	If FIP VLAN discovery fails, driver would stop the discovery and
	will try to use Default VLAN ID 1002 for FCoE traffic.
	BNX2FC driver will keep on altering between these two approach until a
	VLAN associated with FCoE traffic is discovered. BNX2FC driver
	has a timeout of 20 seconds for each approach.

